Path Reconstruction and Interconnection Modeling (PRIM)

ABSTRACT

Internet data such as Border Gateway Protocol routing information and traceroute measurements are processed to create realistic predictive models of the paths IP traffic is likely to take between any two points on the Internet, even when direct measurements of the paths is not feasible. The prediction includes three categories: topology (what paths may exist), weighting (which paths are more or less likely to be taken under varying operational circumstances), and performance (latency, loss, jitter, etc. across the predicted paths).

INCORPORATION BY REFERENCE; DISCLAIMER

Each of the following applications are hereby incorporated by reference:application Ser. No. 15/260,957 filed on Sep. 9, 2016; application Ser.No. 13/896,888 filed on May 17, 2013; application 61/648,977 filed onMay 18, 2012. The Applicant hereby rescinds any disclaimer of claimscope in the parent application(s) or the prosecution history thereofand advises the USPTO that the claims in this application may be broaderthan any claim in the parent application(s).

BACKGROUND

Any point (server, laptop computer, a network-ready device, a wirelesshand-held device, etc.) can establish a communication link via theInternet with another point and data can be exchanged between these twopoints. The exchanged data can be a short text message, a document, or amulti-media object, and even streaming audio/video. It is difficult toknow how traffic may flow between two selected points on the Internet ata given time. In an ideal situation, we would simply log into one of theendpoints, run a traceroute to the other location (or perform a similarprocedure), and observe the path traffic takes. Then we would repeat theexperiment in reverse to observe the opposite (and not necessarilysymmetric) path. Typically, however, one cannot log into just anyInternet point and run such measurements.

The data traffic from one endpoint to another generally flows viasystems (routers, cables, wireless links, etc.) of one or moreAutonomous Systems (AS). A pair of points on the Internet maycommunicate with each other via one or more AS's. When the communicationoccurs via two or more AS's, a physical (electrical) connectivitybetween those AS's is required but is usually not adequate; acontractual relationship must also exist between those AS's

An estimate of the path to be taken by traffic between two points can,however, be useful in several ways. A user may be able obtain estimatesof various traffic characteristics such as average delay, delay jitter,reliability of the connection, etc. Furthermore, in some situations auser at one endpoint can choose between feasible alternate paths to theother endpoint. That decision can be informed by the knowledge of theexpected traffic characteristics, that can change from time to time.Some techniques allow for estimation of various traffic characteristicswhen an actual, router-level path between the points of interest isprovided. These methods are not applicable, however, when the actualrouter-level paths are not provided. Therefore, there is a need for atechnique that can efficiently predict a path between two endpoints onthe Internet. There is also a need for techniques that can estimatevarious traffic characteristics without relying on the knowledge ofactual router-level paths.

SUMMARY

Various embodiments of the present invention facilitate accurateprediction of a likely path between two specified endpoints on theInternet. The path can be predicted at the AS level and also at a routerlevel. If an AS-level path is computed or obtained otherwise,characteristic of the router-level path can also be determined. This isachieved by, in part, modeling the connectivity of the two endpoints. Ingeneral, there are three phases to this predictive process: identifyingthe paths along which traffic can reasonably flow (topology),identifying the paths along which we believe most traffic will flow(weighting), and predicting the performance characteristics of thosepaths (congestion, latency, packet loss, jitter).

In the end, we can use these models to rank and rate rival paths acrossthe Internet between two points, or between two prefix sets (regions ofInternet address space), for purposes of helping enterprise customersmanage their planetwide Internet connectivity more intelligently.Network operations staff within the enterprise can use PRIM'spredictions to compare their actual experience (the paths their trafficactually takes through the Internet) with the ideal experience supportedby the “best” path available through a given provider or set ofproviders.

Strategic planners within enterprise IT can use these techniques tolearn what paths are possible, and which of these paths are desirable(“fit for purpose” for their particular application environment, whetherthat be voice, video, file transfer, financial trading, or any otherInternet-delivered service). They can identify and quantify the pathsavailable through their current provider, and compare them to competingpaths available through other feasible future providers. These modelingtechniques can help strategic planners manage end-to-end connectivityvariants as they would an investment portfolio, understanding therelative strengths, weaknesses, and risks of various wide-areaconnectivity alternatives in combination.

There may be other applications of such models beyond the immediateapplications in network operations and strategic planning, includingrisk assessment, insurance and other actuarial applications, compliancemonitoring, cybersecurity, business intelligence, market analysis, thepreparation of what-if scenarios, and the assessment of networkimpairment due to disaster or attack.

Accordingly, in one aspect a method, an article of manufacture thatincludes instructions stored thereon, and/or a system facilitatesreconstruction of a path between two endpoints. The reconstructionincludes performing at a server generation of a set of connectivitymodes based on border control gateway protocol (BGP) messages receivedvia an interface. Each connectivity mode represents an autonomous system(AS)-level path between a first endpoint and a second endpoint. A usermay be interested in sending data from the first (source) endpoint tothe second (destination) endpoint, and may also receive data at thefirst endpoint from the second endpoint. For each one of theconnectivity modes in the set, a pivotal region that includes at leastone AS is identified.

The reconstruction also includes generating, based on routing data, anumber of feasible physical paths, each of which represents arouter-level path between a proxy to the first endpoint and a proxy tothe second endpoint. From the number of feasible physical paths, afrontier set is determined, and the frontier set includes at least onerouter interface. A connectivity mode from the set of connectivity modesis selected, and a router interface in the frontier set is identifiedsuch that the identified router interface is determined to be associatedwith at least one of the ASs in the pivotal region of the selectedconnectivity mode. A router-level path associated with the identifiedrouter interface is designated as a plausible path.

In some embodiments, an AS-level path representing a connectivity modeincluded in the set of connectivity modes is generated, at least inpart, by determining a transit relationship and/or a peeringrelationship between a pair of ASs. An edge from a transit provider to atransit customer may be preserved in the AS-level path, while any edgefrom the transit customer to the transit provider may be excluded. Insome embodiments, the AS-level path includes at the most one pair ofAS's such that the relationship between the two AS's of the pair isdetermined to be a peering relationship. Even though two AS's may appearto be in communication with each other, the exclusion of an edge from acustomer to a provider and/or restricting the peering AS's as describedabove, enables accurate modeling of AS-level paths by taking intoconsideration BGP and other routing policies of these AS's.

In some embodiments, if an announcement of a BGP origination of a routeto a network prefix corresponding to an identified router interface isreceived from at least one of the ASs in the pivotal region of aconnectivity mode, the identified router interface is determined to beassociated with that AS.

A connectivity mode may be selected, at least in part, by analyzingobservations of BGP reachability between a pair of AS of a candidateconnectivity mode during a predetermined period. If the observedreachability is at least equal to a preselected threshold that candidatemode is selected, and otherwise the candidate mode is rejected. The BGPreachability can indicate how frequently paths between a pair of AS's onthe AS-level path representing the candidate mode are announced and/orwithdrawn, and can thus be a measure of reliability of the candidatemode.

For example, in one embodiment, a connectivity mode is selected, atleast in part, as follows: For each connectivity mode in the set ofconnectivity modes, BGP reachability between the first and secondendpoints along the AS-level path representing the mode, observed duringa predetermined period, is analyzed. A first probability of connectionfrom the first endpoint to the second endpoint along the AS-level pathis computed, and a second probability of connection from the secondendpoint to the first endpoint along the same AS-level path is alsocomputed. A connectivity mode is selected based on the computedprobabilities. For example, a probability in one direction, i.e., fromthe first to the second endpoint or from the second to the firstendpoint is maximized, or the mode is selected such that a simple orweighted combination of the two probabilities is maximized.

In some embodiments, the selection of a connectivity mode comprisesweighing each connectivity mode in the set of connectivity modes basedon BGP data received from several BGP observers. A BGP observer istypically a source that monitors BGP data, providing a BGP perspective.Each connectivity mode may be weighted based on reachability of one ormore BGP peers/observation points that are a direct or indirect customerof that AS or service provider of which the first and/or secondendpoints are also direct or indirect customers.

In some embodiments, path reconstruction further includes selecting theproxy to the first endpoint. The proxy selection includes determiningfrom BGP data a prefix associated with the first (e.g., source)endpoint, and designating a responding host within the prefix as theproxy to the first endpoint. As both the proxy and the first endpointshare a common prefix, a path from/to the proxy is likely to be similaror representative of a path from/to the first endpoint. The proxyselection may also include determining from BGP data a prefix associatedwith the second (e.g., destination) endpoint, and designating aresponding host within that prefix as the proxy to the second endpoint.

The determination of the frontier set may include receiving router-leveldata collected at a number of collectors. For example, a first path froma first collector to the proxy to the first endpoint is received. Asecond path from a second collector to the proxy to the second endpointis also received, and a router interface associated with both the firstand second paths is identified. The determination of the frontier setmay include selecting a pivotal region, and selecting an entry pointinto the pivotal region and an exit point from the pivotal region. Ashortest path from the entry point to the exit point may be identified,and router interfaces that are not present on the shortest path may beremoved from the frontier set. The first collector may be different thaneach of the first endpoint, the second endpoint, the proxy to the firstendpoint, and the proxy to the second endpoint. The second collector mayalso be different than each of the first endpoint, the second endpoint,the proxy to the first endpoint, and the proxy to the second endpoint.In some embodiments, the first collector is also different than thesecond collector, while in other embodiments the second collector is thesame as the first collector.

In some embodiments, path reconstruction further includes estimatingperformance of the plausible path. The estimated performance may includeone or more of a delay, jitter, reliability, and availability.Estimation of the performance includes, for a connectivity mode in theset of connectivity modes, computing a transition probability that themode is replaced by another mode in the set. The transition probabilitymay be computed using BGP data. The mode selection may be based, atleast in part, on transition probability. By taking into considerationthe transition probabilities, the estimated performance can represent anaggregate performance of two or more plausible paths associated withdifferent likely modes.

In some embodiments, the plausible includes a number of routerinterfaces, and the performance estimation includes analyzing aninstance of a router timing model, an instance of an edge timing model,or both. The instance of the router timing model may correspond to arouter interface within the several of router interfaces, and canaccount for a processing delay within the corresponding routerinterface. The instance of the edge timing model may correspond to anedge between a pair of adjacent router interfaces within the severalrouter interfaces, and can account for a transition delay involved inexchanging data between the two router interfaces in the pair. Therouter timing model may include a statistical timing model based on, atleast in part, historically observed performance values at a routerinterface. The edge timing model may also include a statistical timingmodel based on, at least in part, historically observed performancevalues at an edge between a pair of adjacent router interfaces.

In some embodiments, the performance estimation further includesbuilding an aggregate timing model. The aggregation may include firstand second instances of the router timing model, and the first andsecond instances may correspond to first and second router interfaceswithin the number of router interfaces. Alternatively or in addition,the aggregation may include first and second instances of the edgetiming model, and these first and second instances may correspond to anedge between a first pair of router interfaces and an edge between asecond pair of router interfaces, respectively. The first and secondpairs may be selected from the number of router interfaces. Additionallyor alternatively, the aggregation may include the first instance of therouter timing model and the first instance of the edge timing model.Thus, the aggregated model can account for performance of one or morerouters interfaces forming a path, performance of one or more edgesbetween the router interfaces, and/or performance of a combination ofone or more router interfaces and one or more edges.

In some embodiments, performance estimation further includes updatingthe instance of the router timing model based on, at least in part, ameasured delay at the router interface associated with that instance.The instance of the edge timing model may also be updated based on, atleast in part, a measured delay at the edge associated with thatinstance.

In another aspect, a method, an article of manufacture that includesinstructions stored thereon, and/or a system facilitates estimation ofintra-mode paths, which includes

receiving in memory at least a partial autonomous system (AS)-level paththat includes a first AS and a second AS. The partial and/orendpoint-to-endpoint AS-level path corresponding to a mode ofcommunication or a connectivity mode between a first (e.g., source)endpoint and a second (e.g., destination) endpoint. The estimation ofintra-mode paths also includes selecting a first hand off pair thatincludes an exit point from the first AS and an entry point into thesecond AS. The exit point is in communication with the entry point, sothat traffic can be delivered from the first AS to the second AS at theexit point-entry point pair.

The selection of the exit and entry points is based on, at least inpart, one or more of: (i) a property of the first AS, (ii) a property ofthe second AS, and (iii) a statistical property of traffic that passesvia the first AS and is associated with a proxy to the first endpointand/or a proxy to the second endpoint. The estimation of intra-modepaths further includes identifying a set of router-level paths withinthe first AS, from a first entry point into the first AS to the exitpoint from the first AS within the first hand off pair. Thus, trafficmay flow within the first AS from the first entry point thereof to theexit point in the first hand off pair. Thereafter, the traffic may behanded off to the second AS, e.g., for ultimate delivery to the second(e.g., destination) endpoint.

The property of the first and/or second AS includes one or more of: (i)a number of router-level hop counts within the first AS on a path fromthe first entry point into the first AS to the exit point from the firstAS within the first hand off pair, (ii) a geographic distance betweenthe first entry point into the first AS and the exit point from thefirst AS within the first hand off pair, and (iii) a number of points ofpresence associated with both the first AS and the second AS.

The selection of the exit and entry points may include maximizing afirst distance within the first AS. The first distance may include oneor more of: (i) a number of hop counts on a path to the exit point fromthe first AS within the first hand off pair, and (ii) a geographicdistance between the first entry point into the first AS and the exitpoint from the first AS within the first hand off pair. The selection ofthe exit and entry points may include, additionally or in thealternative, minimizing a second distance from the exit point from thefirst AS within the first hand off pair to the proxy to the secondendpoint. The second distance may include one or more of: (i) a numberof hop counts on a path from the exit point from the first AS within thefirst hand off pair to the proxy to the second endpoint, and (ii) ageographic distance between the exit point from the first AS within thefirst hand off pair and the proxy to the second endpoint. When the firstdistance is maximized and the second distance is minimized, the handlingof the traffic by the first AS is maximized. The first distance may alsobe minimized, maximizing the second distance, such that the handling ofthe traffic by the first AS is minimized.

In some embodiments, the selection of the first hand off pair includesselecting a candidate exist point from the first AS, and selecting acandidate entry point into the second AS. The candidate exit and entrypoints are selected such that a frequency of traversal of trafficbetween the candidate exit point and the candidate entry point isgreater than a selected threshold. The selected candidate exit point isdesignated as the exit point within the first hand off pair, and theselected candidate entry point is designated as the entry point withinthe first hand off pair. The frequency of traversal of traffic mayinclude a frequency of traffic directed to the proxy to the firstendpoint and/or the proxy to the second endpoint. The frequency oftraversal of traffic may also include, alternatively or in addition, afrequency of traffic received from the proxy to the first endpointand/or from the proxy to the second endpoint. Thus, the frequency oftraversal of traffic can correspond to the frequency of any trafficflowing between a candidate exit point and a candidate entry point, orto the frequency of traffic that is related to the proxy to the firstendpoint, proxy to the second endpoint, or both.

In some embodiments, the estimation of intra-mode paths further includesselecting a router-level path from the identified set of router-levelpaths such that a path metric corresponding to the selected router-levelpath satisfies a specified threshold. The path metric may include one ormore of a frequency of traversal of traffic via the selectedrouter-level path, delay, jitter, reliability, and availability.

One or more router-level paths in the identified set may each include anintermediate router node, and the path metric may include a frequency oftraversal of traffic via an intermediate router node. A path may beselected if most traffic, as indicated by the frequency of traversal,flows via an intermediate router node on that path. The frequency oftraversal of traffic via the intermediate router node may include afrequency of traffic directed to/from the proxy to the first endpoint,the proxy to the second endpoint, or both. Thus, the frequency oftraversal of traffic can correspond to the frequency of any trafficflowing via the path, or to the frequency of traffic that is related tothe proxy to the first endpoint, proxy to the second endpoint, or both.

In some embodiments, the estimation of intra-mode paths further includesselecting a second hand off pair. The second hand-off pair includes anexit point from the first AS and an entry point into the second AS, andthe exit point is in communication with the entry point. The selectionof the second hand-off pair is based on, at least in part, one or moreof: (i) a property of the first AS, (ii) a property of the second AS,and (iii) a statistical property of traffic that passes via the first ASand is associated with the proxy to the first endpoint and/or the proxyto the second endpoint. The estimation of intra-mode paths furtherincludes identifying a set of router-level paths within the first AS,from a second entry point into the first AS to the exit point from thefirst AS within the second hand off pair. The second entry point intothe first AS may be the same as the first entry point into the first AS.

In the following sections, we describe these modeling techniques inthree parts, starting with topology (predicting the existence of paths),moving on to weighting (predicting the relative popularity of paths) andfinally to performance (predicting congestion, latency, packet loss,jitter, and the like).

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the sameparts throughout the different views. Also, the drawings are notnecessarily to scale, emphasis instead generally being placed uponillustrating the principles of the invention.

FIG. 1 schematically depicts various modes according to one embodiment;

FIG. 2A schematically depicts feasible router level partial pathsaccording to one embodiment;

FIG. 2B schematically depicts a likely router level partial paths basedon different modes, according to one embodiment;

FIG. 3A schematically depicts selection of entry and exit pointsaccording to one embodiment; and

FIG. 3B schematically depicts selection of intra-AS paths according toone embodiment.

DETAILED DESCRIPTION I. Predictive Internet Topology

The first challenge is to identify and map out the likely alternativesfor arbitrary end-to-end physical topology. The initial goal is tosimply compute a set of plausible paths that are consistent with BGProuting and observed traceroute dynamics—distinguishing the possiblefrom the impossible—while deferring the refinement of relativeprobabilities of the alternative modes we identify until the next phaseof analysis.

To construct a model of the plausible physical paths between a pair ofendpoints, we employ an algorithm that incorporates BGP pathinformation, relationship tags, router identification, and rawtraceroute data. The process of topology derivation breaks down intofour basic challenges:

-   -   1. Enumerate and classify the primary connectivity modes between        the endpoints in question, starting with BGP routing data and        our inter-provider relationship classification database.    -   2. Use the global graph formed by the union of traceroute        adjacencies seen during some observation period to enumerate        subgraphs of physical paths toward each of the endpoints.    -   3. Identify the most plausible frontier along which to unify        subtraces into meaningful end-to-end paths.    -   4. Aggregate the resulting set of modes and paths to simplify        presentation and analysis; for example, by combining paths        through multiple interfaces on the same router chassis.

Step 1. Identifying Connectivity Modes

To begin the process, we note that three different kinds ofinter-provider relationships may contribute to the predictedconnectivity modes between two distant providers.

First are the universally visible transit edges in the global BGP graph:provider-customer relationships upstream of the endpoints. Over thecourse of the time period in question, the source and destinationprefixes are observed to have transit relationships, directly ortransitively, with a set of NSPs. For example, the originating ASN of aDSL customer may buy transit from a regional provider such as Pacnet,who in turn buys transit from Cogent and Level3.

In addition to provider-customer transit relationships, some of theproviders who serve one endpoint, or the other, but not both, may haveknown peering relationships. In many situations, an endpoint is servedby a single AS, but in some situations, a complex endpoint (such as aset of servers, a large organization, or a city), may be served by morethan one AS's. Economics dictates that a valid connectivity mode,represented by a plausible end-to-end AS path, should ideally be“valley-free”; that is, it should contain at most one transit providerwhose customer cone includes both endpoints, or contain exactly onepeering adjacency between transit providers whose customer cones eachinclude one of the endpoints. This “most central” provider or pair ofpeered providers we call the pivot region of the AS path, for reasonsthat will become clearer when we begin to synthesize traceroute-likepaths that conform to the AS path's adjacency constraints.

The final complication in deriving connectivity modes comes in the formof what we call occult edges in the connectivity graph: additionalrelationships between providers that are not visible in BGP, but thatare visible as unexplained adjacencies between router interfaces in thefull traceroute dataset. These may represent missing hops in thetraceroute data, or they may represent legitimate peering relationshipsthat are simply not visible in BGP without a direct peering relationshipwithin the customer cone of one side or the other. We make these occultedges available for path generation, treating them as an unexposed BGPpeering edge, and seeing whether the resulting paths are consistent withmeasured traces.

With transit edges, peering edges, and occult edges in place, we canfinally build a set of potential end-to-end AS paths that can supportconnectivity between the endpoints. We call each of these paths aconnectivity mode, because in traditional traceroute studies, theytypically give rise to one or more identifiable modes in thedistribution of end-to-end round trip latencies.

With reference to FIG. 1, an EndPoint L 102 is associated with anautonomous system AS1, 104. The EndPoint L 102 is also associated withanother AS, AS2 106. By observing the BGP updates, it is determined thatAS1 104 has transit relationships with AS3 108 and AS4 110. AS2 106 hasa transit relationship with AS5 112. These AS's have transit and peeringrelationships with various other AS's, as depicted in FIG. 1. In generalas described herein, the observation that one AS has a transit orpeering relationship with another AS implies that such a relationshipwas determined to exist by a processor by monitoring and analyzing BGPupdates over a certain time period such as an hour, a day, a month, orseveral months. EndPoint R 148 is associated with AS 146 which hastransit relationships with each of AS's 138, 140, 144, 142.

The AS 128 has a transit relationship with AS's 116 and 134. FIG. 1depicts that AS 116 can exchange data with EndPoint L 102 via otherAS's. Similarly, AS 134 can exchange data with EndPoint R 148.Therefore, AS 128 is designated as an AS within a pivotal region 150. As126 is similarly designated as another AS within the pivotal region 150.AS's 122, 124 have a peering relationship, and AS 122 can exchange datawith the EndPoint L 102 while AS 124 can exchange data with EndPoint R148. Therefore, AS's 122, 124 are also included within the pivotalregion 150.

Accordingly, several modes exist between EndPoint L 102 and EndPoint R148. For example, a first mode includes an AS-level path AS1 104, AS3108, AS 114, AS 120, AS 126 in the pivotal region 150, AS 132, AS 136,AS 144, and AS 146, for sending data from EndPoint 102 to EndPoint R148. An AS-level path for receiving data at EndPoint L 102 from EndPointR 148 would include the same AS's in the first mode. Alternatively, asecond mode including an AS-level path AS 146, AS 140, AS 134, AS 128 inthe pivotal region 150, AS 116, AS4 110, and AS1 104 is feasible forreceiving data at EndPoint L 102 from EndPoint R 148.

It should be understood that FIG. 1 is illustrative only and that ingeneral any one AS may have a transit and/or peering relationships withas few as one other AS or several (4, 10, 20, 50, etc.) AS's. Therefore,several, i.e., tens, hundreds, thousands, or more modes are feasiblebetween any two endpoints. Various embodiments of the system describedherein are configured to identify relationships between a pair of AS'sbased on BGP updates, and to determine several AS-level paths (or modes)between two endpoints.

Step 2. Interpreting Connectivity Modes as Physical Paths

The next step is to gather traceroute data (sequences of routerinterfaces) from a large, diverse pool of collectors, and build it intoa connectivity graph for the entire Internet. We maintain a livepopulation of over a million responding host interfaces, scatteredthroughout the entire routed Internet. On a continuous basis, we gathertraces to each of these targets from a diverse population of dozens ofcollectors. From this set of raw traces, we create a master directedgraph G, whose nodes are all observed router interfaces on earth, andwhose edges are all observed interface adjacencies. G is maintained on adaily basis, and is available for inspection on any day in history sincethe beginning of our collection activities.

The next step in synthesizing a model of connectivity between twoarbitrary sets of IP-numbered endpoints is to mine G for informationabout the reachability of the source and destination sets of IPaddresses in question (for convenience, call these two sets L and R, asin “left” and “right”). For example, L might be the IP address of a DSLcustomer, and R the set of IP addresses used by a popular web servicesuch as Facebook. Using BGP data for the observation period in question,we identify one or more most-specific routed prefixes containing theendpoints in L and R, as well as its originating autonomous system(s).

We then identify the set of live traceroute target hosts within therouted prefixes within L and R, and within every other prefix originatedby the same autonomous systems. We slice global graph G to extract apair of directed graphs G_(L) and G_(R) whose nodes are routerinterfaces and whose edges are observed interface adjacencies seen enroute to targets in L or R.

Finally, we identify the “frontier set” of router interfaces common toG_(L) and G_(R): the set of shared global router interfaces that appearin at least one trace to some host on each side. These interfaces arepotential pivot points that can be used to construct end-to-end physicalpaths between L and R.

This frontier set will typically be huge, on the order of thousands ofnodes for two widely separated networks with diverse connectivity. Itwill include many interfaces within the core of the shared ASNs thatprovide service to both sides, only a few of which will actually beplausible points of traffic exchange for a path between L and R. It willalso include many interfaces that have nothing to do with reachabilityof L or R, by virtue of being topologically important for the Internetconnectivity of one or more of our many traceroute collectors.

Step 3. Establishing Percolation

The final step uses the routing logic of identified connectivity modesbetween L and R to winnow the wheat from the chaff in the frontier set.For each predicted connectivity mode, we identify pivotal frontiersubsets (router interfaces that are common to traces toward L and R inG) that are routed within most-specific prefixes originated by some ASwithin the mode's pivot region. In general, the identified routerinterface (in the frontier set) is associated with one of the AS's inthe pivotal region because it: has an IP address; that IP address iscontained within a network prefix; and the prefix has been announced inBGP by an AS in the pivotal region. If such subsets exist, we take theset of end-to-end physical paths between L and R through these points asour model for end-to-end connectivity.

For example, if both L and R are in the customer cone of Level3(AS3356), and if we have identified a set of common router interfaceswithin Level3 that are seen en route to both L and R in traces from ourcollectors around the world, then those common interfaces constitute apivotal frontier subset, and we can use them as the midpoints ofsynthetic paths between L and R through Level3. If the frontier subsetis still very large, we may select a further subset of these commoninterfaces that meet some selection criterion, such as appearing on theshortest path between the entry and exit points within Level3.

If there are no non-empty pivotal frontier subsets, we make one attemptto bridge an intra-ASN gap by injecting edges from another subset ofglobal G: the traces to each side of the connectivity mode's pivotregion. For example, if we can identify a short, well-supported pathbetween the European and North American sides of a provider's backbonenetwork, then we can use that intra-ASN path to complete the modeledend-to-end connectivity mode for which that provider is pivotal, but forwhich we see no common interfaces in traces to the endpoints.

Specifically, when there are no frontier routers within the pivot AS,i.e., the pivotal region, or if there are too few frontier routers(perhaps because the frontier routers we have identified are not onshortest and/or statistically most likely paths between the most likelyentry/exit points for the pivot AS), we can synthesize paths through thepivot AS just as we would synthesize a path across any non-pivot AS, asdescribed below with reference to FIGS. 3A and 3B. In particular, wepick hot potato, cold potato, and/or statistically well-supported pathsthat draw on the large pool of router interfaces observed in the largeruniverse of traces through the pivot AS, even though those routerinterfaces and/or the edges between them may not actually have beenobserved en route to one or the other endpoint.

When the frontier set is “too full” of common routers that were seen enroute to both endpoints (i.e., their respective proxies), we can choosethe routers in the pivotal frontier subset with respect to the mostlikely entry and exit points for the pivot ASN. The most likely entrypoint to the pivot AS is usually determined together with the mostlikely exit point from the previous AS, as described below, e.g., withreference to FIGS. 3A and 3B, because they are in direct communicationand form a crossing point into the pivot AS. The most likely exitpoint(s) from the pivot AS are also selected in a similar manner. Then,we identify those elements of the frontier set that are on thehigh-likelihood paths between the selected entry and exit points, and wecan recreate/synthesize one or more end-to-end router-level pathspassing through these frontier routers within the pivot AS.

Thus, if we can identify a well-supported, high-likelihood path thatincludes routers from the frontier set that are determined to reach bothendpoints, we can generate router-level paths from the observedconnections. If we cannot identify such paths through an observedfrontier set, we infer/determine the “bridge” across the pivot ASN.Typically, the pivotal AS tends to be a very busy global provider,across which we have numerous measurements to use as raw data forunderstanding and reconstructing the patterns of traffic flow.

With reference to FIG. 2A, the EndPoint L 102 has two proxies—Proxy L1202 and Proxy L2 204 within the AS 104. The EndPoint R 148 has a proxy R206. Proxy L1 202 is in communication with Proxy R 206 via severalrouters 210. Traceroutes to each of these proxies may be performed atone or more collectors (typically tens, hundreds, or even thousands ofcollectors). For the sake of clarity, the collectors are not depicted inFIGS. 2A and 2B. In some instances, the proxies would allow thetraceroute to run while the actual endpoint may not. In other instances,the collectors may not be aware of a specific requested interface (e.g.,an endpoint), and hence, a traceroute to that endpoint may not beavailable. A traceroute to a suitable proxy to that endpoint, however,can be used in this situation. The traceroute data can reveal variousrouter-level paths between the two proxies 202, 206, in both directions.The routers 226, 228, for example, may be encountered in a traceroute tothe Proxy L1 202 but not in a traceroute to the Proxy R 206. Thus, therouters 226, 228 are determined not to be useful (not likely to beencountered) in reaching the Proxy R 206 form the Internet at large fromplaces (e.g., routers) other than Proxy L1 202. Based on the traceroutedata, it is observed, however, that each of the routers 212, 214, 216,218 is in communication with both Proxy L1 202 and Proxy R 206.Therefore, these routers are included in the frontier set 250, and therouters 226, 228 are not included. In general, a frontier set includesor consists essentially of the routers that have been seen on the way toproxies to both endpoints. It should be understood that in somesituations, a proxy may be the actual endpoint of interest.

By identifying the AS corresponding to each router in the frontier set250, it is determined that the router 216 is within the AS 128 and thatthe router 218 is within the AS 126. As described with reference to FIG.1, the AS's 128, 126 are included in the pivotal region 150. Thus, therouters 126, 128 form the pivotal frontier subset 252. Once the routersin the pivotal frontier subset are determined, various router-levelpaths corresponding to a certain mode can be determined. For example,with reference to FIG. 2B the mode (i.e., AS-level path) that includesthe AS's 104, 110, 116, 128 a router-level path exists from Proxy L1 202and routers 228, 226, 224, 222, 216. Another partial router-level paththat includes the routers 202, 228, 226, 230, 220, 218 corresponds to adifferent mode that includes the AS's 104, 110, 114, 120, 126.

For the sake of convenience, only a part of the router-level path fromthe Proxy L1 202 up to the pivotal frontier subset 252 is depicted inFIG. 2B. In general, end-to-end router-level paths exist between the twoproxies associated with two end points, in both directions. Also for thesake of convenience, all AS's in the pivotal region 150 are not depictedin FIGS. 2A and 2B. One or more routers may exist within those AS's(such as AS's 122, 124) and other router-level paths may be identifiedto/from Proxy L1 202, Proxy L2 204, and Proxy R 206. FIG. 2B depictsthat the pivotal region 150 is included within or encompassed by thefrontier set 250. In some embodiments, however, the pivotal region andthe frontier set may only partially overlap or may not overlap at all.If the pivotal region and the frontier set do not overlap, the pivotalfrontier subset is initially empty, but routers to be included in thepivotal frontier subset can be identified as described above.

A pivotal region can be associated with a selected mode, in which casethe pivotal frontier subset corresponds to that mode. The pivotal regioncan also be associated with more than or all modes (such as the pivotalregion 150), and in that case the pivotal frontier subset may beassociated with more than one modes, such as the pivotal frontier subset252. Finally, each end point may be associate with one, two, or moreproxies and any AS on an AS-level path may include one, two, or morerouters. In fact, an AS typically includes tens, hundreds, or even morerouters, some or all of which may be identified as part of therouter-level paths.

Step 4. Simplification and Aggregation

We take a few additional steps to simplify the resulting connectivitygraph for presentation and interpretation. Recall that our global graphG includes weights for each edge, and that these weights are computedindependently for every traceroute target prefix supported by a givenedge. The subgraphs supporting L and R, which we merge along thefrontier, allow us to prune poorly-supported nodes and subpaths that aredominated by more popular alternative paths within the same provider,exposing the most likely paths. We also merge multiple target nodes in Land R that have common prefix membership and routing. Finally, we mergetransit nodes that we can identify as alternative interfaces on the samerouter.

II. Predictive Path Weighting

The final graphs generated by the topology prediction phase, one foreach connectivity mode, can be thought of as a set of plausiblealternative physical connectivity options between L and R. Which mode isactually used for communication from L to R, and back again, is likelyto be different (path asymmetry), and will vary over time.

We next describe various techniques for predicting which modes, andwhich paths through each mode graph, will give rise to the most likelypaths traffic will actually take in each direction between two arbitraryendpoints.

Step 1: Mode Selection

The first step is to identify the relative probabilities of selectionfor each connectivity mode (each based on an observed or inferred BGP ASpath from end to end) in each direction between the endpoints ofinterest.

We observe that BGP reachability of the two endpoints varies throughoutthe course of the day, as local routing announcements change. One way ofpredicting the probability of each connectivity mode in each directionbetween the endpoints at a particular moment is to take into account thespecific BGP announcements last seen by a particular set of BGPobservers (peering routers throughout the Internet) at a given moment.The observation of each BGP observer/peer, i.e., the BGP data collectedby that BGP observer/peer forms, at least in part, a perspectivethereof.

Integrating Global BGP Peer Perspectives

For example, at a given moment, we may survey each of several hundredBGP sources, examining and integrating the most recent BGP updates (orwithdrawals) from that source to determine the FIB (ForwardingInformation Base) entry corresponding to the endpoint IP prefixescorresponding to L and R. By surveying the “opinions” of the best routes(and backup routes not taken) at each of these BGP observation points,we can arrive at an indirect determination of the ASN-level pathpreferences held by representative points within the routing systemworldwide.

We can use these observations directly to establish global path weights(for example, 40% of BGP peers worldwide (i.e., the BGP observationpoints) have a best route through Cogent to one of the endpoints. Cogentis an example of an AS—typically a level 1 or level 2 AS which could bein the pivotal region. Another 30% have a best route through Level(3),another 20% through Singapore Telecom, and so forth). Level(3) andSingapore Telecom are also examples of AS similar to Cogent.

Selecting Similar BGP Peer Perspectives

Alternatively, we can examine the subset of global BGP peers whose ownconnectivity most closely resembles that of the counterparty endpoint.For example, if endpoint A is inside the customer cones of Level(3) andCogent, we may more highly weight the best route preferences of our BGPpeers that are also within the customer cones of Level(3) and/or Cogentwhen predicting the most popular routes that endpoint A might take toreach endpoint B.

Either way, by combining relevant best-path observations from across ourBGP peering set (including policy-signaling factors such as AS pathprepending), and by taking into account the behavior of the BGPprotocol's decision process and tie-breaking procedures on the actualroutes announced by the originating autonomous system for each endpoint,we can predict and weight the most likely connectivity modes from amongthose identified as potential candidates in the topology predictionphase.

Out of Band Policy Information

We may also use out-of-band information about the providers in question,or data learned from external sources such as flow measurements, tomanually steer the calculation of mode probabilities. This isparticularly important when considering the probability that a givenmode will be used for outbound connectivity. In practice, this decisionoften includes establishment of a single preferred outbound provider (a“default route”) for cost or other reasons; a default route is notnormally exposed to direct measurement by either inbound traceroutes orannounced BGP routes.

Our own outbound traceroute measurements from a large, diverse set ofcollection locations can be mined for insight into such policy decisionsunder a range of transit provider diversity conditions. For example, wecan use our own traceroute collector population to estimate thelikelihood that an endpoint with Level(3) and Cogent transit will simply“point default” at the Cogent link, causing the Cogent mode(s) to bedisproportionately selected for connectivity to a broad range of remotetraceroute targets.

Integrated Mode Weighting

Finally, having integrated all of these sets of observations, wegenerate a “directional mode weighting” for each of the potentialconnectivity modes identified by the topology prediction phase. That is,we can assign distinct probabilities of selection for each connectivitymode on paths from L to R, and from R to L (the return path). Theseprobabilities can either be time-integrated (over the course of a periodof observation, such as a day), or instantaneous (determined by routingand other information integrated through time to a particular singleinstant at which the prediction is valid).

Step 2: Intra-Mode Path Selection and Mode Splitting

Having identified the modes (AS paths) along which traffic may flow, andhaving weighted the probability of selection of a particular mode in aparticular direction between the endpoints, the next step is to weightthe likelihood of selection of the various potential router-level pathsthat support the given connectivity mode.

In practice, this reduces to the challenge of determining the likelypaths through each provider's network along the path, and determiningthe most likely points at which providers may hand off traffic to eachother in each direction along the path.

Predicting Inter-Provider Handoffs

The methodology for choosing among the various entry/exit paths betweentwo global carriers is based on heuristics and rule-based reasoning.Pure graph-theoretic strategies such as shortest path, maximal support,etc. are unlikely to reproduce longstanding conventions and bestpractices, such as cold-potato interdomain routing among settlement-freepeers. Moreover, such practices virtually guarantee the existence ofasymmetric physical paths between the endpoints, even within the“single” connectivity mode represented by one end-to-end AS path amongmany.

To integrate these variants into our path calculations, we consider eachmode as a “generator” for a set of physical paths in each direction,each of which is consistent with the overall ASN-level muting associatedwith the mode in question, but which can be weighted and consideredseparately in terms of the relative probability of adoption for use fromeach endpoint.

Graph-Based Path Calculations

For example, when evaluating competing paths across a peeringrelationship between two cooperating providers, we can use a min-maxcalculation (in which we attempt to find an inter-provider handoff pointthat maximizes time and/or hopcount on one provider's network, whileminimizing the time and/or hopcount spent on the other provider'snetwork en route to the endpoint. Depending on which leg is minimizedand which maximized, this combination of constraints allows the model tosimulate the primary kinds of inter-provider exit strategies (hot- andcold-potato routing) encountered in settlement-free peering adjacencies.

Observed Traversal Count Support

We can also take into account the relative frequency of traversal ofdifferent combinations of entry/exit points between two providers inorder to rank probable inter-provider handoff points in support of agiven connectivity mode. Our global graph G stores the weights (edgetraversal counts) observed over time for each edge, en route to eachtarget prefix on Earth. We can therefore consider the relativepopularity of different inter-provider handoff sites, integratingobservations for one target prefix, or for a larger set of targetprefixes such as those originated by a given autonomous system or withina given geographic region, depending on the endpoints.

Predicting Intra-Provider Path Selection

Another strategy for path weighting involves the use of edge weightingfrom our original traceroute set to determine distinct paths through asingle provider's core, and estimate their relative selectionprobabilities in various scenarios. This allows us to integrate andexamine the total weighting on edges supporting various paths through Gin support of a given connectivity mode.

In many cases, these edge traversal counts can reveal that traffic to asingle prefix or set of prefixes (geographic region, ISP, etc.) mayfavor a limited number of entry and exit points within a givenprovider's transit network, and therefore predict as more likelyintra-provider paths connecting those points. When there are multipledistinct paths through a single provider's network, we can ofteneffectively split a single connectivity mode predicted by BGP intomultiple connectivity modes, corresponding to these distinct paths, eachwith a modeled weight of selection.

For example, on a global carrier's network, there may commonly be a“short path” and a “long path” towards a given customer: one correspondsto the shortest, most direct route (which may be expensive and/orcongested) and the other to the indirect route going the long (wrong)way around the Earth. We can use our destination-based edge weightingsto estimate the relative probabilities that each of these sub-modes willbe used in practice to reach a particular destination through particularexit points, given the fact that traffic has reached the carrier'snetwork at a particular entry point. The relative probabilities of thesepaths may take other factors into account; for example, the time of dayand the paths predicted for other destinations served by the sameprovider.

Feeding Back Performance Data

Finally, we can impose additional “common sense” heuristic constraintson the paths selected within a given provider, and betweeninterconnected providers. For example, we can incorporate latency datafrom the next phase, or geolocation data about the particular routersparticipating in each path, to downweight as unlikely those paths thatincorporate undesirable predicted performance or geographic wandering.

For example, we may exclude as unlikely multiple ocean-crossing routinghops from North America to Europe and back again between North Americanendpoints. Or we may downweight circuitous paths whose total modeledround trip latency is substantially longer than the average latencies onother available paths. Or we may penalize paths across edges or throughrouter interfaces that are believed to be congested (detected fromdiurnal loading patterns in latency or packet loss in the next phase).

We know that all of these suboptimal routing conditions do occur on theInternet in real life, but they tend to elicit customer complaints andfind resolution fairly quickly, making them less likely to be among thepredicted paths on an average day.

In general, the system attempts to determine the most likelyrouter-level path that traffic will take within an autonomous system, toa handoff point with another autonomous system. This requires inferenceof the real-world routing policies that have been agreed upon betweenthe two autonomous systems in question. In practice, these policies arenot publicly announced and can vary widely. These policies can bedetermined by monitoring and analyzing BGP data. Once a policy isdetermined, entry and exit points of an AS and the intra-AS paths can bedetermined.

For example, with reference to FIG. 3A, the hand off policies include“hot potato routing” (exit at the first opportunity). The hand offoccurs between AS 302 and AS 304, which are neighbors on a AS-level pathfrom a source endpoint to a destination endpoint. A router 312 is anentry point into AS 302. According to hot potato routing, an exit point314, which is “nearest” to the entry point 312 is selected. The exitpoint 314 hands off traffic to an entry point 316 in AS 304, forsubsequent delivery to the end destination. In this context, “nearest”can mean having the shortest hop count, shortest actual distance,shortest delay, etc.

If the hand off policy is determined to be “cold potato routing” (exitas “close” to the ultimate destination as possible), the hand off mayoccur at another exit point 318 instead of at the exit point 314. Inthis case, traffic is handed off to router 320 of AS 304. “Close” forthese purposes can be router-level hop count, geographic distance inmiles, or city-level points of presence traversed (e.g., number of largecities along the path, where the two autonomous systems may have theopportunity to exchange traffic). Providers often select this strategyif they have a well-provisioned network and want to keep the traffic aslong as possible to maximize quality of service. Once the traffic hasbeen handed to the next ASN, the provider loses control over how thetraffic is treated. In this embodiment, the hopcount within the next ASis not necessarily minimized, but the distance between the handoff andthe ultimate destination (which may or may not be within the AS to whichhand off is made) is minimized. Again the “distance” metric can include,for example, remaining router hop count, remaining geographic distance,or remaining cities to be traversed. Other more selective agreementsgoverning specific kinds of traffic and specific points of interchangemay also be determined from the BGP data. For example, traffic of acertain kind (e.g., voice traffic, video traffic, and traffic destinedfor a specific geographic location) may be handed off via exit point 322in AS 302 to the entry point 320 in AS 304.

In one embodiment, the model for path determination can be selected inadvance; e.g., we may assume hot-potato routing (the most common casebetween peers), based on a peering relationship that was determined toexist based on BGP data. In another embodiment, we infer the policybased on previously collected traceroutes through a certain AS to othertargets (e.g., proxies of destination endpoints) whose autonomous systemorigination and/or geography is similar to the selected destinationendpoint. The statistics and/or inferences are usually drawn from tracesdirected toward the traffic targets (i.e., the endpoints), since allInternet policies are generally concerned with getting traffic closer toits destination. Alternatively, or in addition, the statistics and/orinferences can be based on the AS properties and/or traceroutes from asource (e.g., a proxy of a source endpoint) whose autonomous systemorigination and/or geography is similar to the selected source endpoint.

Different approaches can also be combined; for example, we can think ofeach pair of connected routers within an AS as having been observed witha given frequency in general (for all traces, for all traffic traversingthe AS, over a long period of recorded time). For example, withreference to FIG. 3B, the path from the entry point 312 to the exitpoint 314 via nodes (e.g., routers) 332, 334 may be taken at frequencyf1. The path between the same entry and exit points via nodes 336, 334may be taken with frequency f2; and via nodes 336, 338 may be taken withfrequency f3. In addition, a path from the entry point 312 to the exitpoint 318 via nodes 336, 338 may be taken with frequency f4 and the pathbetween the nodes 312, 318 via nodes 340, 338 may be taken withfrequency f5. Moreover, we also consider the frequencies of linksbetween neighbor nodes such as 336, 334; 336, 338, 340, 338, etc. We canselect as more likely a path to an exit point that corresponds to amaximum of the average recorded traversal frequencies across the path'sconstituent edges. In general, the likelihood of selecting a path is afunction of the relative weights (frequencies of traversal) of the edgesthat form the path. These weights are typically derived from thetraceroute observations. The path selection may also be directed by therelative weights of only a subset of edges/routers along the path.

We can think of each pair of connected routers within an AS as havingbeen observed with a given frequency in a particular destinationsituation (restricting our analysis to the subset of traces whosedestination is the destination is the destination endpoint, or proxy ofthe destination endpoint, or whose destination is within the samegeographic region (city, country) as our second destination, or both).In the example discussed above, the frequencies f1 through f5 aregeneral frequencies. Instead, we can consider frequencies s1 through s5,where s1 is the frequency at which the path between the nodes 312, 314via nodes 332, 334 is taken for traffic to be ultimately directed to aspecified destination endpoint. The other frequencies, s2 through s5 canbe similarly computed by taking into consideration the traffic destinedfor the selected destination endpoint. We can then select among paths asin the previous case, but using the situational frequencies instead ofthe general frequencies to select the most likely paths and thereforethe most likely exit/handoff points.

Similarly, we can use the frequencies of edge traversal based onobservations collected within a limited time range, rather than usingall the traces collected over a long time range. For example, we canrestrict our analysis to recent history (traces from the last hour, orthe last day) since paths change over time. Or we can restrict ouranalysis to a particular recurring time window (the same time of dayover the course of a week, the same day of the week over the course of ayear) because traffic exchange policies may have a seasonalcomponent—for example, traffic may flow differently between providersduring the busiest hours, in the early evening local time, and duringthe lightly loaded weekend days.

We generally assess the complete end-to-end path between the twoendpoints, and that includes performing the above-described analyses intwo ways—once forward, using the destination (second) endpoint to guidepath selection from the source (first) endpoint, and again backwards,using the source (first) endpoint to guide the path selection back fromthe destination (second) endpoint. These two forward/backward pathsbetween the endpoints are often not the same. Hot potato routing, forexample, easily generates path asymmetry, as each side of thecommunication wishes to find and preferentially exploit the nearest exitpoint to the other.

III. Predictive Path Performance

After topology prediction and path weighting, the third PRIM challengeis the estimation of the performance characteristics of our inferredend-to-end paths, based on partial observations of the performance ofindividual network segments.

We approach this challenge in two ways: at the connectivity mode level(essentially, predicting the variability of AS paths between twoendpoints based on the BGP routing history of either) and predictingpacket loss, jitter, and round trip latency between endpoints based onthe most probable physical (router-level) paths between them.

Step 1: Connectivity Mode Stability

We can associate transition probabilities between members of the set ofplausible connectivity modes that PRIM predicts. In each case, thereceipt of a BGP UPDATE message containing the withdrawal of a key route(the transit path through a given provider at one endpoint, for example)can render some connectivity modes unavailable, and cause others to beselected.

We can predict the next-most-probable backup path that would be selectedin the absence of the withdrawn route, once global routing hasconverged, and in so doing, assign a transition probability to eachother mode that may replace it. Changes in routing affecting oneendpoint may affect inbound connectivity from the other endpoint, orthey may affect the outbound path as well (for example, when thepredicted default route for outbound traffic goes away as the providerconnection is lost).

Similarly, when a new route becomes available (for example, because atransit provider connection has been restored, or because the endpointin question has gained a new transit provider) we can model theprobability that connectivity in- and outbound will shift to takeadvantage of the newly accessible modes.

Based on the intraday history of the routing stability of the endpoints,we proceed to construct a model of the stability of each connectivitymode: the probability that traffic in flight will be forced to takedifferent paths within a short time. Continuous path change, can createconditions for poor performance of TCP-based applications such as voiceand video, especially across fiber-optic runs at intercontinentaldistances, where the bandwidth-delay product is high. This is especiallytrue when the predicted performance of competing paths variessignificantly (potentially resulting in out-of-order delivery of packetswithin the application's traffic stream).

Step 2: Path Stability and Performance

Within a particular connectivity mode, we can also model the end-to-endperformance of a predicted physical path between two endpoints. Toaccomplish this, we integrate a large number of individual observationsof the round-trip latencies between our traceroute collectors and eachindividual router interface (or “hop”) along the paths to all of ourworldwide traceroute targets.

Node and Edge Performance Prediction

With these measurements, we start by creating a model of the latencyacross each edge in the global graph G (the time taken for packets totravel between two connected router interfaces). This model may takeinto account the delay across each router (the time added by each nodein the graph, a function of buffer depth and queuing delay) as aseparate distribution from the latency across each connection (the timeadded by each edge in the graph, dominated by speed-of-light concerns).These measurements may be performed using traces to both proxies from alarge set of collectors. In particular, a collector performs ameasurement at a router along the way en route to a proxy. In otherwords, a traceroute measurement contains timing measurements to each ofthe router-level hops along the way from a collector to a target (e.g.,proxy). The relative timing of the measurements to two connected routerinterfaces gives us the timing statistics for the “edge” between thosetwo router interfaces. The model of each router interface/edge can beupdated based on additional measurements and/or feedback, and from theupdated models, ultimately the aggregate model of an entire path thatincludes all the connected edges and router interfaces can also beupdated.

We can distinguish the two phenomena (the delay associated with anode/router interface and the delay associated with an edge between tworouters interfaces), by the characteristics over time of theirindividual contributions. The speed-of-light contribution from edgedelay is most typically modeled as a fixed mode (the physical distanceacross most links does not vary appreciably; nor does the transmissionmedium) with some narrow spread due to measurement error. The queuingdelay at each router contributes a quite different (often heavy-tailed)variable delay due to congestion, which can create visible time-of-dayand day-of-week sinusoidal dependencies.

Path Performance Prediction

Working with a set of performance models for individual nodes and edgesin graph G, we can iteratively refine the models by predicting theaggregate path performance of known (measured) end-to-end paths, andthen feeding back corrections to the individual models.

For example, the difference between measured time and modeled time canbe divided among the contributing edges according to some proportionalrule, and fed back as a model correction. Through a large number ofiterative corrections, the models for performance across each edge andthrough each node will tend to converge on the true distributions forlatencies (assuming that these true distributions are appropriatelystationary over the period of observation).

As a refinement, rather than naively assuming path symmetry, we can takeinto account the most probable return paths generated by our previousPRIM phase, thus supplying a more accurate model for the invisiblereturn path upon which each measured traceroute depends. Since theend-to-end performance of a traceroute path is just as often determinedby the characteristics of the return path as by the visible forwardpath, this allows us to generate significantly more accurate models ofthe latency distributions across nodes and edges using the availablelatency measurement data.

Path Integration Methods

Even when accurate models of individual edges and nodes are not possiblefrom the data (for example, because two nodes are always seen insequence, making it difficult to untangle their individual contributionsto latency) we can still compute models for the connected nodes orentire subpaths as a group, with the same kind of composite latencydistribution.

In general, we can derive latency distributions for any path or set ofpaths through G by performing convolution of successive individuallatency distributions along the paths. At a branching point (two or morepotential paths) we scale down each distribution by the weightedprobability that the given path is taken, as determined by our previouspath-weighting methodology; at a merge point (two or more paths comingtogether at a common interface) we add the incoming latencydistributions. In this way we can derive complex composite latencydistributions for the paths between two endpoints, taken as a whole; wecan also derive simpler latency distributions for each contributing pathbetween the endpoints.

When direct measurements are available, we can compare these models tothe distributions observed and feed back corrections to previous modelphases; for example, the observed latency distribution may cause us toadjust our model of the weighted probability that a given path is taken(because the measured latencies associated with that path are not inevidence as a predicted mode in the end-to-end latency distribution).

It should be noted that in these computations, the path taken intoconsideration is a router-level path between proxies to the endpoints ofinterest, and the path is based on a connectivity mode, e.g., anAS-level path between the two endpoints. As such, the estimatedperformance of the path is expected to be a close approximation of theperformance of an actual path between the two endpoints.

Some known systems can measure or estimate performance of an actual pathbetween the endpoints of interest. As such these known systems requireaccess to at least one of the actual endpoints, and hence, cannot beused when such access is unavailable. In this context, access requiresthe administrative ability to run traceroute from the node (e.g., anendpoint), or otherwise to perform active measurement of theconnectivity and performance of the network paths that are reachablefrom that node. The techniques we described above provide estimateswithout requiring access to any endpoint of interest.

Some known systems can measure or estimate performance of paths betweenproxies to the endpoints. Unlike the techniques we described above,however, the estimates according to known systems are not based ontiming models. Instead, these measurements or estimations are baseddirectly on measurements and computations performed at at least one ofthe proxies. Even though access to a proxy may be available in general,it may not be available when the performance estimate is required. Also,the paths for which the performance is determined according to the knownsystems are not selected according to one or more connectivity modes(AS-level paths). The model based system does not require measurementsfrom or to an endpoint.

Additional Applications

We can use the modeled performance of particular paths to inferadditional information about the nodes and edges that support thosepaths. For example, we may use commonalities in latency distributions toconclude that two router interfaces are in the same geographic region,or even in the same facility.

Or we may use information about the modes in the performancedistribution to infer the details of connectivity; for example, we mayinterpret particular long latency modes as indicating that particularlinks are implemented as LEO or geosynchronous satellite hops, eitherwith symmetric connectivity or with an asymmetric return path providedover terrestrial connectivity.

We can use our latency models to predict the “fitness for purpose” of agiven path to support the connectivity requirements of a given class ofapplications; for example, a high-latency path may support asynchronousfile transfer and email, but may be inappropriate for supporting voiceand video applications. A path with low average latencies, but highvariance, may be particularly problematic for real-time applicationswith low jitter tolerance.

Together, the PRIM methodologies allow us to characterize the pathstraffic may take through the Internet between two endpoints of interest,whether those paths are real (consistent with observation) orcounterfactual (available by purchasing a new transit connection orestablishing peering with a new partner). Paths and their performancecan be predicted for a broad range of Internet counterparties, scalingfrom single IP endpoints to network prefixes, service providers, andeven entire geographic regions. PRIM path analysis allows us toconstruct “what-if” scenarios, modeling the changes in connectivity andapplication performance that would result if a key connection weredisrupted, or if a key network performance bottleneck were resolved. Inmany cases, it allows us to identify points of potential congestion anddisruption (“chokepoints”) representing significant risks to a givenconnectivity pattern, so that they may be defended against.

PRIM allows us to rate and rank the available modes of connectivity, andphysical paths supported by those modes, so that a customer can quantifyand audit the benefits and drawbacks of their available connectivity toglobal Internet counterparties. Key provider dependencies, potentiallyproblematic or fragile connectivity modes, and the underlying causes ofWAN performance issues are among the business issues that can be exposedby the application of the PRIM modeling techniques to an enterprise'sInternet connectivity.

In our testing, PRIM has successfully distinguished among probable pathsto many globally distributed networks (Hotmail Singapore versus HotmailUS, for example) from arbitrary points of interest. It has alsosucceeded in localizing many of the provider-provider handoffs thatcarry traffic out of a given ISP at an ASN level; that is, the AS pathvisible in a manual traceroute from an IP address within that ISP isusually one of the connectivity modes identified by PRIM. Within thesemodes, PRIM's predictions of the cities and countries traversed—and inmany cases, the specific routers encountered along the way—are often inclose agreement with empirical observations.

Key inputs to the process (the global connectivity graph G, the map ofoccult ASN adjacencies not evident in BGP, the “edge tag” relationshipsthat distinguish peering from transit, the graph of transit supportrelationships for each routed prefix, the measurements of latency toeach traceroute target and each intermediate hop, and theinterface-level maps of individual provider core networks used toconstruct bridging approximations) are all products with a shortshelf-life, thanks to the dynamic nature of Internet relationships.

Caution must be therefore be used in applying predicted paths far fromthe original time of calculation, in a nonstationary global environmentin which underlying routing or policy conditions may have shiftedsignificantly.

It is clear that there are many ways to configure the system components,interfaces and methods described herein. The disclosed methods andsystems can be deployed on convenient processor platforms, includingnetwork servers, personal and portable computers, and/or otherprocessing platforms. Other platforms can be contemplated as processingcapabilities improve, including personal digital assistants,computerized watches, cellular phones and/or other portable devices. Thedisclosed methods and systems can be integrated with known networkmanagement systems and methods. The disclosed methods and systems canoperate as an SNMP agent, and can be configured with the IP address of aremote machine running a conformant management platform. Therefore, thescope of the disclosed methods and systems are not limited by theexamples given herein, but can include the full scope of the claims andtheir legal equivalents.

The methods and systems described herein are not limited to a particularhardware or software configuration, and may find applicability in manycomputing or processing environments. The methods and systems can beimplemented in hardware or software, or a combination of hardware andsoftware. The methods and systems can be implemented in one or morecomputer programs, where a computer program can be understood to includeone or more processor executable instructions. The computer program(s)can execute on one or more programmable processors, and can be stored onone or more storage medium readable by the processor (including volatileand non-volatile memory and/or storage elements), one or more inputdevices, and/or one or more output devices. The processor thus canaccess one or more input devices to obtain input data, and can accessone or more output devices to communicate output data. The input and/oroutput devices can include one or more of the following: Random AccessMemory (RAM), Redundant Array of Independent Disks (RAID), floppy drive,CD, DVD, magnetic disk, internal hard drive, external hard drive, memorystick, or other storage device capable of being accessed by a processoras provided herein, where such aforementioned examples are notexhaustive, and are for illustration and not limitation.

The computer program(s) can be implemented using one or more high levelprocedural or object-oriented programming languages to communicate witha computer system; however, the program(s) can be implemented inassembly or machine language, if desired. The language can be compiledor interpreted.

As provided herein, the processor(s) can thus be embedded in one or moredevices that can be operated independently or together in a networkedenvironment, where the network can include, for example, a Local AreaNetwork (LAN), wide area network (WAN), and/or can include an intranetand/or the Internet and/or another network. The network(s) can be wiredor wireless or a combination thereof and can use one or morecommunications protocols to facilitate communications between thedifferent processors. The processors can be configured for distributedprocessing and can utilize, in some embodiments, a client-server modelas needed. Accordingly, the methods and systems can utilize multipleprocessors and/or processor devices, and the processor instructions canbe divided amongst such single or multiple processor/devices.

The device(s) or computer systems that integrate with the processor(s)can include, for example, a personal computer(s), workstation (e.g.,Sun, HP), personal digital assistant (PDA), handheld device such ascellular telephone, laptop, handheld, or another device capable of beingintegrated with a processor(s) that can operate as provided herein.Accordingly, the devices provided herein are not exhaustive and areprovided for illustration and not limitation.

References to “a microprocessor” and “a processor”, or “themicroprocessor” and “the processor,” can be understood to include one ormore microprocessors that can communicate in a stand-alone and/or adistributed environment(s), and can thus can be configured tocommunicate via wired or wireless communications with other processors,where such one or more processor can be configured to operate on one ormore processor-controlled devices that can be similar or differentdevices. Use of such “microprocessor” or “processor” terminology canthus also be understood to include a central processing unit, anarithmetic logic unit, an application-specific integrated circuit (IC),and/or a task engine, with such examples provided for illustration andnot limitation.

Furthermore, references to memory, unless otherwise specified, caninclude one or more processor-readable and accessible memory elementsand/or components that can be internal to the processor-controlleddevice, external to the processor-controlled device, and/or can beaccessed via a wired or wireless network using a variety ofcommunications protocols, and unless otherwise specified, can bearranged to include a combination of external and internal memorydevices, where such memory can be contiguous and/or partitioned based onthe application. Accordingly, references to a database can be understoodto include one or more memory associations, where such references caninclude commercially available database products (e.g., SQL, Informix,Oracle) and also proprietary databases, and may also include otherstructures for associating memory such as links, queues, graphs, trees,with such structures provided for illustration and not limitation.

References to a network are not limited to the full Internet, and caninclude portions thereof. References herein to microprocessorinstructions or microprocessor-executable instructions, in accordancewith the above, can be understood to include programmable hardware.

Unless otherwise stated, use of the word “substantially” can beconstrued to include a precise relationship, condition, arrangement,orientation, and/or other characteristic, and deviations thereof asunderstood by one of ordinary skill in the art, to the extent that suchdeviations do not materially affect the disclosed methods and systems.Further, references herein to real-time can be understood to beabbreviations for “substantially in real-time.” Although the illustratedembodiments of the methods and systems refer to certain aspects being in“real-time,” such aspects may be provided in other manners.

Throughout the entirety of the present disclosure, use of the articles“a” or “an” to modify a noun can be understood to be used forconvenience and to include one, or more than one of the modified noun,unless otherwise specifically stated.

Although the methods and systems have been described relative tospecific embodiments thereof, they are not so limited. Obviously manymodifications and variations may become apparent in light of the aboveteachings.

Many additional changes in the details, materials, and arrangement ofparts, herein described and illustrated, can be made by those skilled inthe art. Accordingly, it will be understood that the methods and systemsprovided herein are not to be limited to the embodiments disclosedherein, can include practices otherwise than specifically described, andare to be interpreted as broadly as allowed under the law.

Accordingly, we claim:

What is claimed is:
 1. A method comprising: obtaining a plurality ofround-trip latency measurements between each of a plurality of routerinterfaces in the computer network and each of a plurality ofcollectors; generating a graph based on the round-trip latencymeasurements, the graph having a plurality of nodes and a plurality ofedges, each node in the plurality of nodes representing a correspondingrouter interface in the plurality of router interfaces and each edge inthe plurality of edges representing a corresponding path between a pairof router interfaces in the plurality of router interfaces; creating amodel of latency across each edge in the plurality of edges; andpredicting performance of a path between the first endpoint and thesecond endpoint based on the model of latency, wherein the method isperformed by at least one device including a hardware processor.
 2. Themethod of claim 1, wherein obtaining the plurality of round-trip latencymeasurements comprises measuring a round-trip latency between a firstcollector in the plurality of collectors and a first router interface inthe plurality of router interfaces, the first router interface disposedalong a path between the first endpoint and the second endpoint.
 3. Themethod of claim 1, wherein obtaining the plurality of round-trip latencymeasurements comprises measuring timing to each router-level hop betweenthe first endpoint and a collector in the plurality of collectors. 4.The method of claim 1, wherein creating the model of latency comprises:accounting for delay associated with each edge in the plurality ofedges; and accounting for delay associated with each node in theplurality of nodes.
 5. The method of claim 4, wherein accounting for thedelay associated with each edge in the plurality of edges comprisesassigning a fixed propagation delay to each edge in the plurality ofedges.
 6. The method of claim 4, wherein accounting for the delayassociated with each node in the plurality of nodes comprises assigninga variable queuing delay to each node in the plurality of nodes.
 7. Themethod of claim 6, further comprising: varying the variable queuingdelay as a function of time.
 8. The method of claim 1, wherein creatingthe model of latency comprises performing a convolution of successiveindividual latency distributions along a path between two nodes in theplurality of nodes.
 9. The method of claim 1, wherein creating the modelof latency comprises weighting different paths at a branching point inthe graph.
 10. The method of claim 1, wherein creating the model oflatency comprises adding incoming latency distributions at a mergingpoint in the graph.
 11. The method of claim 1, further comprising:adjusting the model of latency based on a measurement of actual latencyalong a path in the computer network.
 12. The method of claim 1, furthercomprising: estimating a geographic proximity of two router interfacesin the plurality of router interfaces based on the performance of thepath between the first endpoint and the second endpoint.
 13. The methodof claim 1, further comprising: predicting a fitness of the path betweenthe first endpoint and the second endpoint for at least one ofasynchronous file transfer, email, voice applications, videoapplications, or real-time applications based on the performance of thepath between the first endpoint and the second endpoint.
 14. One or morenon-transitory computer readable media comprising instructions which,when executed by one or more hardware processors, causes performance ofoperations comprising: obtaining a plurality of round-trip latencymeasurements between each of a plurality of router interfaces in thecomputer network and each of a plurality of collectors; generating agraph based on the round-trip latency measurements, the graph having aplurality of nodes and a plurality of edges, each node in the pluralityof nodes representing a corresponding router interface in the pluralityof router interfaces and each edge in the plurality of edgesrepresenting a corresponding path between a pair of router interfaces inthe plurality of router interfaces; creating a model of latency acrosseach edge in the plurality of edges; and predicting performance of apath between the first endpoint and the second endpoint based on themodel of latency.
 15. The one or more media of claim 14, whereinobtaining the plurality of round-trip latency measurements comprisesmeasuring a round-trip latency between a first collector in theplurality of collectors and a first router interface in the plurality ofrouter interfaces, the first router interface disposed along a pathbetween the first endpoint and the second endpoint.
 16. The one or moremedia of claim 14, wherein obtaining the plurality of round-trip latencymeasurements comprises measuring timing to each router-level hop betweenthe first endpoint and a collector in the plurality of collectors. 17.The one or more media of claim 14, wherein creating the model of latencycomprises: accounting for delay associated with each edge in theplurality of edges; and accounting for delay associated with each nodein the plurality of nodes.
 18. The one or more media of claim 14,wherein creating the model of latency comprises performing a convolutionof successive individual latency distributions along a path between twonodes in the plurality of nodes.
 19. The one or more media of claim 14,wherein the operations further comprise: adjusting the model of latencybased on a measurement of actual latency along a path in the computernetwork.
 20. A system comprising: at least one device including ahardware processor; the system being configured to perform operationscomprising: obtaining a plurality of round-trip latency measurementsbetween each of a plurality of router interfaces in the computer networkand each of a plurality of collectors; generating a graph based on theround-trip latency measurements, the graph having a plurality of nodesand a plurality of edges, each node in the plurality of nodesrepresenting a corresponding router interface in the plurality of routerinterfaces and each edge in the plurality of edges representing acorresponding path between a pair of router interfaces in the pluralityof router interfaces; creating a model of latency across each edge inthe plurality of edges; and predicting performance of a path between thefirst endpoint and the second endpoint based on the model of latency.